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1. STATUS OF CLAIMS 

Claims 11 to 29 are pending in this application. All claims have been 
finally rejected. All claims have been rejected under 35 U.S.C. 103(a). Claims 24 and 25 
have been rejected under 35 U.S.C. 112(2). No claim has been allowed, and all of the 
claims are under Appeal. 

2. STATUS OF AMENDMENTS 

Since the Examiner has not indicated that any of the amendments filed in 
the response dated December 23, 2003 have not been entered, it is assumed that all 
amendments filed have been entered. 

3. SUMMARY OF THE INVENTION 

The present invention relates to a method for providing audio and visual 
communication between a speaker and a hearing impaired person. The first step of the 
method is to provide the speaker with a headset frame having a camera attached 
thereto and positioned to capture images of the mouth of the speakerA hearing impaired 
person is provided with a display, wherein sound emitted by the speaker and received 
by the hearing impaired person essentially does not allow the hearing impaired person 
to understand the speaker. Continuous video images of the mouth of the speaker using 
the camera. The images are transmitted in real-time in an uncompressed manner to the 
display for the hearing impaired person to view such that movement of the mouth of the 
speaker coincides with sound emitted by the speaker and enables the hearing impaired 
person to understand the speaker. 

It should be appreciated that for the present invention, the person skilled in 
the art is one knowledgeable in not only the art of video-conferencing, but also in 
technology used for hearing-impaired persons. The existing technology in this field is 
very deficient because technologies used for regular users are not being adapted to the 
hearing-impaired properly by taking into account the specific needs of the hearing- 
impaired. 
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4. ISSUES 

There are four issues outstanding in this application. 

The first issue is whether the term "real time" used in claim 1 1 should be 
interpreted as within a time span of less than 33 milliseconds. 

The second issue is whether US patent 5,742,335 to Cannon teaches 
"real time" transmission as understood in issue (1). 

The third issue is whether it is an invention to use, for the hearing- 
impaired, equipment that is known in the art for the hearing-enabled. 

The fourth issue is whether there is motivation to modify the equipment 
described by US patent 5,886,735 to Bullister to enable communication with a hearing- 
impaired person. 

5. GROUPING OF CLAIMS 

Claim 11 is the only independent claim and claims 12 to 29 are all 
dependent directly or indirectly on claim 11. The claims under Appeal are directed to 
one invention with the dependent claims specifying more specific limitations. For the 
purposes of simplifying the Appeal, all claims should be taken as one group. 

6. ARGUMENTS 
First Issue 

To qualify issue (1), the claims as originally filed have the term "real-time". 
While the Applicant has defined in the application the following: 

"According to the person skilled in the art of the present invention, 
real time is defined as a maximum delay of 33 msec (milliseconds) 
between the moment when the image is perceived and the moment 
when the sound is perceived (c.f. The Effect of Imperfect Cues on the 
Reception of Cued Speech", written by Maroula Sarah Bratakos of 
the Massachusetts Institute of Technology, September 1995)." (page 
3, lines 7-12) 

the Examiner refuses to read the term in the claim as meaning 33 msec or less and 
argues that the specification is not the measure of the invention. The Examiner 
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maintains that limitations contained in the specification cannot be read into the claims 
for the purpose of avoiding the prior art. 

The Applicant believes that according to section 2111 of the MPEP, 1-the 
claims must be given their plain meaning unless thev are defined in the specification , 2- 
"plain meaning" refers to the ordinary and customary meaning given to the term by 
those of ordinary skill in the art , and 3-the Applicant may be own lexicographer. 

Section 2111 of the MPEP states that claims must be given their broadest 
reasonable interpretation. However, In re Morris, 127 F.3d 1048, 1054-55, 44 USPQ2d 
1023, 1027-28 (Fed. Cir. 1997), the court stated that "the PTO applies to verbiage of the 
proposed claims the broadest reasonable meaning of the words in their ordinary usage 
as they would be understood by one of ordinary skill in the art, taking into account 
whatever enlightenment by way of definitions or otherwise that may be afforded by the 
written description contained in applicant's specification." (emphasis added). From this 
statement, it can be understood that any definition clearly and explicitly stated in the 
description is to be taken into account when giving a term in a claim the broadest 
reasonable meaning. 

Section 2111 of the MPEP also states that "the broadest reasonable 
interpretation of the claims must also be consistent with the interpretation that those 
skilled in the art would reach". The Applicant submits that the definition of 33 msec or 
less was given in accordance with a publication relating to real time for cued speech. As 
indicated above, the field of the present invention is technology used for hearing- 
impaired persons and a person skilled in this field would readily understand that a time 
delay above 33 msec would not be possible for a device used for the specific 
application described in the present invention, namely communication with a hearing- 
impaired person. 

Section 2111.01(1) of the MPEP is very clear in its interpretion of words in 
a claim that are defined in the specification. In re Zletz, 893 F.2d 319, 321, 13 USPQ2d 
1320, 1322 (Fed. Cir. 1989), the court stated that "the words of the claim must be given 
their plain meaning unless applicant has provided a clear definition in the specification". 
The Applicant hereby submits that the definition provided on page 3, lines 7 to 12 of the 
application as originally filed and reproduced herein above constitutes a clear definition 
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as required by the jurisprudence. Therefore, the term "real time" should be interpreted in 
view of this definition. 

Additionally, Section 2111.01(11) of the MPEP states that "If intrinsic 
reference sources, such as dictionaries, evidence from more than one definition for the 
term, the intrinsic record must be consulted to identify which of the different possible 
definitions is most consistent with applicant's use of the terms". It can be appreciated 
that the term "real time" has come to be known as a broad expression meaing 
substantially simultaneously or with no intentional delay. The Merriam-Webster's 
Collegiate Dictionary of real-time is defined as "the actual time during which something 
takes place" and dates back to 1953. Given that this is the age of technology and the 
possibilities for "real time" are constantly evolving, the definition may vary anywhere 
between no noticeable delay in time and a very specific time delay. It is precisely for this 
reason that the Applicant felt the need to define the term in the application in order to 
correspond to the requirements of the invention. A person skilled in the art of the 
present invention would easily recognize the need for this constraint on "real time" given 
the nature of the invention. Therefore, the limitation of 33 msec or less should be read 
into the claim. 

Finally, Section 2111.01(111) stipulates that the Applicant may be his own 
lexicographer. In re Paulsen, 30 F.3d 1475, 1480, 31 USPQ2d 1671, 1674 (Fed. Cir. 
1994), the court stated that an inventor may define specific terms used to describe 
invention, but must do so "with reasonable clarity, deliberateness, and precision" and, if 
done, must "'set out his uncommon definition in some manner within the patent 
disclosure' so as to give one of ordinary skill in the art notice of the change" in meaning. 
The Applicant hereby submits that the definition of "real time", as set out in the 
specification by the applicant, fully complies with the requirements set out by the courts. 
It is sufficiently clear in the specification to any person skilled in the art of the present 
invention. 

Therefore, the definition as set out should be read into the claim. 
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Second Issue 

Issue (2) is directly related to issue (1). Assuming the term "real time" 
were understood as being within a time span of less than 33 msec, Cannon does not 
teach a real time transmission that meets this definition. 

Cannon is related to the detection of defects in buildings and other large 
manmade structures and more particularly to a method and apparatus of structural 
defect detection using visual and infrared cameras. Two cameras, a local infrared 
camera and a local visual-spectrum camera, are mounted on a helmet that is worn by a 
photographer or camera operator. Video signal transmitters coupled to these cameras 
are mounted on the helmet. Video signals are transmitted by the transmitters and 
received by respective signal monitors at a ground station. A succession of frames 
creates a movie describing each area of interest on the surface of the building. The 
helmet may also include an audio head set and transmitter, by which means the 
photographer can give an audio commentary on the area which he or she is inspecting. 

The "real time" transmission described by Cannon should be understood 
from the context of the application as meaning that someone in a remote location can 
view the images and hear the audio commentary at substantially the same time as the 
photographer performing the inspection. This does not correspond to the definition of 
"real time" as per issue (1), which is having a maximum delay of 33 milliseconds 
between the moment when the image is perceived and the moment when the sound is 
perceived. Cannon does not teach capturing an image of the speaker's mouth and is not 
related to the art of communication between a speaker and a hearing impaired person. 
There is no need to synchronize image and sound since there is no lip movement to 
match with the auditory cues. Therefore, Cannon has no reason to suggest transmitting 
images in an uncompressed manner and meeting the requirements of "real time" 
transmission as per the present application. 

Third Issue 

To qualify issue (3), the Applicant believes that it is non-obvious to use, for 
the hearing-impaired, equipment that is known in the art for the hearing-enabled. 
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As stated above, the person skilled in the art of the present invention is 
knowledgeable in technology used for hearing-impaired persons. Enclosed herewith in 
Appendix A is a signed declaration by an expert in the field of adaptive technology for 
the disabled. This person has significant experience working with technology used for 
people with hearing losses, as evidenced by points (3) and (4) of the declaration. 

As per point (5) of the declaration, the expert has testified that "the field of 
adaptive technology is considered to be an industry separate from the industry of 
regular technology". The Applicant respectfully submits that the two fields are not as 
analogous as one may initially believe them to be. There are additional constraints 
imposed on a system due to the limitations of a hearing-impaired person. These 
limitations are not considered by a maker of standard telecommunication equipment 
and therefore, this equipment is not expected to work for the hearing-impaired. 

As per point (6) of the declaration, the expert has never experienced 
himself video-conferencing with proper synchronization of the image with the sound. We 
can understand what is meant by "proper synchronization" by considering point (7) of 
the declaration, where the expert compares the effect of a lack of synchronization for a 
person with a hearing loss, to the feeling a hearing enabled person might experience 
when watching a movie that has been dubbed into another language. Proper 
synchronization is a requirement for the hearing-impaired. Given that an expert in the 
field has never participated in a video-conference which was adequate for a person with 
a hearing loss, we can conclude that there is no expectation from the part of a person 
skilled in the art that standard video-conferencing equipment will enable communication 
for the hearing-impaired. 

As per point (10) of the declaration, the expert believes that standard 
video-conferencing equipment would fail to allow a person having a hearing loss to 
understand a conversation. This goes to the operability of standard equipment for the 
hearing-impaired, as well as to the expectation of a positive result when dealing with 
technology made for the hearing-enabled. 

As per point (11) of the declaration, the expert does not believe that 
standard video-conferencing equipment is the closest prior art to the claimed invention. 
There is no expectation of operability from standard video-conferencing equipment and 
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therefore, a person in the field of adaptive technology does not look towards this field for 
equipment that can be used for the hearing-impaired. 

In view of the above, it is respectfully submitted that it is non-obvious to 
use standard equipment available in the field of telecommunications specifically for the 
hearing-impaired. 

Fourth Issue 

To qualify issue (4), the Applicant maintains there is no motivation to 
modify the equipment described by US patent 5,886,735 to Bullister to enable 
communication with a hearing-impaired person. 

The Applicant has argued that the main reference cited by the Examiner, 
namely US patent 5,886,735 to Bullister, fails to disclose communication with a hearing- 
impaired person. The Examiner has stated that any person, including a hearing- 
impaired person, may view the display as shown in Figure 3 of Bullister. The question is 
not whether a hearing-impaired person can view the display, but rather whether 
communication between the hearing-impaired person and the speaker is enabled. The 
claim currently in dispute is a method claim which defines achieving a result. If this 
result is not achievable using the equipment described by Bullister, this reference fails to 
teach or suggest the claimed invention. 

Bullister teaches a video telephone headset device which includes a video 
camera configured to capture an image of the user's face when the headset is held 
adjacent to the user's head. The camera may be oriented towards an image capture 
mirror located immediately in front of a user's face. The image capture mirror reflects 
the image of the user's face to the camera. The images are captured and transmitted to 
a display in order to provide visual communication between the speaker and another 
party. The system also provides audio communication between the two parties. 

Communication with a hearing-impaired person cannot be enabled with 
the equipment taught by Bullister. In order to enable communication, the sound and 
image must be synchronized to a level that would allow the hearing-impaired person to 
comprehend. As per the expert's declaration, this synchronization is vital to the 
understanding of a hearing-impaired person when lip-reading. Otherwise, the level of 
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understanding is deeply affected and communication is not enabled. The 
synchronization of image and sound in video-conferencing equipment is affected by 
performing operations on the image signal. Such operations can delay the image signal 
and therefore, affect the synchronicity. 

As stated in the background of the invention of the present application, "In 
the Bullister patent, software image correction occurs, which is time consuming, i.e. at 
least about 500 msec for a full image. Furthermore, signal compression through a 
MPEG compressor takes place, which delays the image transfer of approximately 33 
msec, as does the decompression through a MPEG compressor, the latter not being 
shown in the Bullister patent, but being necessary to decompress the image 
compressed by the MPEG compressor. Thus, important delays amounting to up to 
2000% and more of a real time transmission occur with the device shown in the Bullister 
patent." (page 3, lines 12-19). 

Thus, it is respectfully submitted that not only does Bullister not teach or 
suggest transmitting the images in an uncompressed manner, it teaches away from this 
by specifying that software correction (column 3, lines 44-48) and 
compression/decompression of the image is performed by the disclosed device. It is 
virtually impossible for a system that has such enormous time delays to be used by a 
hearing-impaired person in the context of the present application. Since a hearing- 
impaired person is the receiving party for the images, it is necessary that the images be 
transmitted without compression or decompression to ensure that full motion video be 
transmitted without jittering or frame loss and within a minimum time delay. 
Compression and decompression of a signal can have two consequences: (1) delay of 
the signal and (2) the possibility that a frame is dropped or that the frame rate not be 
conducive for full motion video. For the application where the speaker and the receiving 
party are in the same room, the time delay is an important factor such that sound 
received through air corresponds to the lip motion displayed on screen. For the 
application where the two parties are not in the same room, maintaining full motion 
video to ensure that no frames are dropped such that the communication is clear is also 
very important. There would be a significant delay caused by the distortion correction 
and MPEG compressor, as shown in figure 11 of the reference. This significant delay 
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would be enough to confuse a user that is hearing-impaired and prevent proper 
comprehension of what the speaker is saying. Therefore, communication would not be 
enabled. 

As per point (10) of the expert's declaration, a system which corresponds 
to the criteria as described in columm 11, lines 1-4 of Bullister, namely a commonly 
available POTS line (28.8-33 kpbs) or an ISDN line (64 kbps), provides inadequate 
communication for a hearing-impaired person. It is clear from the reference that the 
system described is fully intended to be used with a low bandwidth system. As a matter 
of fact, this is cited as one of the objects of the invention, as stated on column 3, lines 8- 
11. 

There is no motivation to use a better bandwidth to reduce the delays 
between the sound and image since the reference specifically teaches towards reducing 
the transmission bandwidth requirements. Furthermore, there is no motivation to 
remove the software correction step of the system as it is fundamentally linked to the 
central concept of the system. The software correction is used to correct the signal 
received by the camera capturing the facial expressions of the speaker in the reflection 
given by the mirror on the headset, in order to correct for distortions and perspective 
based on the curvature and on the predetermined position of the headset with respect to 
the face. Without this software correction, the image presented to the second party 
would be of lower quality. This would be counter-intuitive and go against the teachings 
of the reference. 

Therefore, there is no motivation to modify the equipment taught by 
Bullister in order to use it for the hearing-impaired. 

Remarks 

The inventive aspect of claim 1 1 comes from the inventor's recognition 
that technology of the prior art is not compatible with hearing-impaired persons due to 
additional constraints. It is precisely this recognition, that the system taught by Bullister 
cannot be used for hearing-impaired persons, that led to the present invention. The 
combination of providing an image of the speaker's mouth and reducing delays in the 
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system to have the sound emitted by a speaker and movement of the speaker's mouth 
on a display coincide is new and non-obvious above and beyond the cited references. 

Additionally, testimony from the expert can vouch for the significant 
advantage provided to people with hearing losses when using the system of the claimed 
invention. Points (9) and (12) of the declaration are specifically directed to the benefits 
of the Audisee™ system, which is the system produced by the Applicant and on which 
the present application is based. The method as recited in claim 11 is performed when 
using the Audisee™ system. This unexpected result and positive reaction by a person 
skilled in the art should be taken into account when considering the patentability of the 
present claims. 

Accordingly, a decision in favor of the acceptability of the claims under 
appeal is respectfully solicited. 


Respectfully submitted, 



Alexandra Daoud 

Agent of Record, Registration No. 55,992 
CUSTOMER NUMBER 020988 
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Declaration 


I, Tom Sheridan, do hereby solemnly declare that: 

(1) I am a citizen of Canada and my residential address is 420 
Huffman Crescent, Edmonton, AB, Canada T5A 4C8 

(2) I am an Adaptive Technology Specialist with the Specialized 
Support and Disability Services (SSDS) at the University of Alberta. 

(3) I am a specialist in assistive technology (technology that helps 
people with disabilities) and have been working with people having disabilities, 
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such as deaf and hard of hearing people, for 19 years. In August, 1999, I joined 
SSDS as the Adaptive Technology Specialist. My responsibilities include 
assessing technology needs of students and staff with disabilities, recommending 
appropriate adaptive technology, and providing training in the use of that 
technology. I manage two computer labs with adaptive technology at SSDS, as 
well as the technology in satellite labs across campus. I serve as a liaison to 
faculties and departments regarding hardware and software compatibility with 
adaptive technology, and advise on future needs and trends. I serve as a 
consultant with the greater community, providing information to agencies, 
hospitals, schools and other post secondary institutions about available 
technology, training and services. I have nineteen years of experience 
evaluating, assessing and training students, as well as professionals, with all 
types of disabilities in the use of adaptive technology. My training also includes 
Braille transcription (Level II), with six years of Nemeth code, and Level IV 
American Sign Language. 

(4) I am a member of the Alberta Consortium of Rehabilitation and 
Assistive Technology (ACRAT), consultant for Occupation Performance Analysis 
Unit, Faculty of Rehabilitation Medicine University of Alberta (OPAU), member of 
Canadian Association for Disabled Skiing (CADS), and a member of Alberta 
Sports and Recreation Association for the Blind (ASRAB). I am an author or have 
co-authored adaptive technology training manuals for on-campus University 
students. These include Assistive Listening Device manuals, Braille Screen 
Reading manuals, Voice Recognition manuals, Augmentative Communication 
manuals and Alternate Access manuals. I have given the following presentations: 
"Introduction Overview to Assessments and Implementation of Assistive 
Technology for Professionals in Post-Secondary Institutions" at the University of 
Alberta. "Scanning the Horizons Global Perspectives on Higher Education and 
Disability" and "Beyond the White Cane" at AHEAD (Association on Higher 
Education and Disability). 
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(5) The field of adaptive technology is considered to be an industry 
separate from the industry of regular technology. In my experience within the 
industry, it is generally recognized that getting something to work for the needs of 
the hard of hearing and deaf is considered an innovation. To adapt and use 
technology created for the hearing and to have it successfully used for the hard 
of hearing and deaf is considered an innovation in this field. 

(6) I am a hearing person and have participated in video-conferencing 
using standard video-conferencing equipment. I have never, in my life-time, 
experienced video-conferencing where the voice and image were properly 
synchronized. Using such equipment, I was able to detect a delay between 
image and audio reception, the delay being visually evident. Additionally, loss of 
frames often caused the image to momentarily freeze until a subsequent frame 
was received. Both of these factors were a nuisance. I have noticed that the 
farther the distance over which the video-conferencing is taking placing, the more 
noticeable the delay between the sound and the images. If the system were set 
up for all parties to be in the same room, or over a very short distance, the delay 
would be smaller and less noticeable. However, standard video-conferencing 
equipment is rarely used over short distances, as this is not its intended purpose. 

(7) Many people, whether hearing or with hearing loss, lip-reading can 
benefit from the use of visual cues are helpful to everyone during communication 
because it facilitates lip-reading. Use of visual cues are helpful to everyone 
during communication. The added value of lip-reading for a hearing person is not 
as significant but will help fill in small gaps if a word or two is missed or rf the 
speaker has an accent that is hard to understand. A common experience where 
hearing people may notice this is when watching a foreign movie that has been 
dubbed into a different language, for example, a Spanish movie dubbed into 
English. The mismatch between the movement of the lips and the sound being 
heard is noticeable and distracting. Since a hearing person can completely 
understand the audio, there is no significant loss in understanding the foreign 
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movie. However, a person with hearing loss will be significantly more affected by 
this lack of synchronicity. 

(8) To the best of my knowledge and belief, a person with a hearing 
loss who has not lost 100% of his hearing ability is greatly aided in his 
comprehension when provided with both audio and visual support. It is known 
that a person with a severe hearing loss will have a rate of recognition for 
consonants that is approximately 30% greater with audio and visual support than 
if only audio cues or only visual cues were present (Erber, N.P. (1972). Auditory, 
visual, and auditory-visual recognition of consonants by children with normal and 
impaired hearing. Journal of Speech and Hearing Research, vol. 15, 413-422). It 
is also known that in an environment where a signal-to-noise ratio is 
approximately 19.5 dB, a person with a hearing loss will get approximately 60% 
more sentences correct with an audio-visual support than with only an auditory 
support (Alison MacLeod and Quentin Summerfield (1990). British Journal of 
Audiology, 1990, 24. 29-43.). However, introducing a noticeable delay between 
the audio and visual support will take away the benefits afforded by the dual 
support system. 

(9) I have worked with students who have hearing loss and am aware 
that they typically benefit understanding what is being said by combining lip- 
reading and facial expressions (for example, raising of eyebrows, frowns, etc) 
with the sounds of speech. In that context, I have observed the benefits on 
students with hearing loss using the Audisee™ system. The primary benefits can 
be attributed to the fact that the Audisee™ system transmits the sound of the 
speaker and the picture of that person's face (especially lips) in a synchronized 
manner. The student can follow the lip movements at the same time they are 
receiving auditory cues. This synchronicity does not exist in the form of video- 
conferencing technology, where there is always a time lapse between the picture 
and the sound. The benefit of the Audisee™ system in the lecture theatre is 
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particularly valuable because, as the professor turns his/her back to the class to 
write on the blackboard, the student still gets the image of the professor's face. 

(10) To the best of my knowledge and belief, it would not be obvious to 
modify a standard video-conferencing system to synchronize image and audio, 
(especially in the case of a system that must compensate for distortions) by 
doing a transformation operation on the signal that represents the video image. 
Moreover, a system using a commonly available POTS line (28.8-33 kpbs) or an 
ISDN line (64 kbps) is not of high enough quality to match the sound with the 
image. This system would fail to allow a person having a hearing loss to 
understand a conversation because the quality of the video on its own would be 
inadequate, the audio on its own would be inadequate, and the combination of 
the audio and video would not fall within the parameters of 33msec. 

(11) To the best of my knowledge and belief, the closest prior art to what 
the Audisee™ system does is having a video overlay of someone signing (i.e., a 
video overlay of an interpreter doing sign language) while a speaker speaks, 
which works efficiently for the person with hearing loss who knows sign 
language. 

(12) l have recommended the Audisee™ system to persons with a 
hearing loss and have seen them use it. I have witnessed the significant 
advantage provided by the Audisee™ system to persons with a hearing loss. 

(13) The statements made above are of my opinion and do not 
necessarily reflect the opinions of my employer, the University of Alberta. 

(14) I, the undersigned, declare further that all statements made herein 
of my own knowledge are true and that all statements made on information and 
belief are believed to be true; and further that these statements were made with 
the knowledge that willful false statements and the like so made are punishable 
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by fine or imprisonment, or both, under 18 U.S.C §1001 of the United States 
Code and that such willful false statements may jeopardize the validity of any 
patent issued for the above-referenced patent application. 


Tom Sheridan 



Date: 


Oct. & - z°°¥ 
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